After several months of not-too-intensive searching for a KVM-hoster with native IPv6 connectivity, Steve hooked me up with a VPS at Bytemark, and I am very pleased. In addition to a properly virtualised KVM instance, I got access to a console server, which allows me to reset the machine, switch kernels, and log in locally . It s exactly what I wanted, but it was hard to find, as everyone else seems to offer only Xen or Vservers, both of which are nearing end-of-life. Thanks Steve, and Bytemark. NP: Porcupine Tree: Up the Downstair

It took us a bit longer than planned, but we are happy to announce the schedule for the Distro Summit at the upcoming LCA2010 conference. We focused on cross-distro aspects, and we hope that you are as excited as we are about the result. The schedule is displayed on the Distro Summit homepage and I best refrain from duplicating it here.

Dear lazyweb: I am in a dreadful situation! I need to secure e-mail communication between five types of users:

• Outlook 2003 users who receive their mail by POP3 from Exchange,
• Thunderbird users downloading mail with POP3,
• Apple Mail users downloading mail with POP3,
• Webmail users (Squirrelmail and Roundcube),
• and a bunch of Unix folks using proper clients.

To make matters worse: there is no common provider or server infrastructure, so the solution must work across providers and mailboxes.

Requirements

• Human error should be anticipated and prevented.
• Mail between all parties must be encrypted (and signed) automatically. If encryption is not possible, the mail must not be sent. This includes the situation where not all mail recipients keys are known;
• Ideally, one can define rules (using wildcards or regular expressions) to enforce certain policies.
• Mail to other parties may be signed, and it would be good to be able to turn this on and off by default;
• The MIME standard should be employed so that the mail body is not altered, and attachments can be seamlessly encrypted too. Existing MIME parts should be encapsulated as children of a new multipart tree;
• Incoming mail should be automatically and seamlessly decrypted/verified, and the user must be alerted if the verification fails.

Ideally, the solution will be open-source. However, if proprietary software performs better, then we will gladly use that where required.

Previous attempts So far, we ve tried (and were let down by)

• GpgOL, which has very irky and brittle integration with Outlook and can only do inline signing/encryption.
• S/MIME, which seems supported by all involved clients, but Outlook does not really allow you to specify policies. Encryption seems to be opportunistic at best, which is not enough.
• GPGrelay, which is promising because transparent, but apparently messes with MIME, and I am unsure whether it can fit between Outlook and Exchange. I have yet to run real tests though. If you are a GPGrelay user, or you d like to try it (it s Windows-only), please get in touch.
• WinGEAM, but that apparently no longer exists, and neither is the underlying GEAM.

Help! Does anyone have proper suggestions over what to use? In fact, if you are interested in devising a solution, or even deploying it, there s money to pay you for those services. Write in if you are interested.

I accompanied Penny to Prague, from where she embarked into the mountains today for a week-long Moodle developer meeting. We froliced in the city a bit, tried to avoid the hordes of anonymous tourists that flooded the city, steered clear of the Christmas kitsch that was all over, sampled Czech food wherever we could, and enjoyed the local beer. On Friday, Petr Baudis took us out to The Pub, an ingenious concept by the Czech brewery Pilsner Urquell: every table has taps from where one can draw beers without waiting or having to get up, guests accumulate a tab measured in litres, and a huge screen shows which The Pub instance has the best beer throughput. Add to that an automated ordering system for salty snacks, and the brewery ensures a ready beer flow with need for no more than two staff members. Penny and I had already decided that Pilsner Urquell was our favourite Czech beer, mostly due to its bitterness. We also sampled the local Staropramen, and the well-known Budvar, and found our preference reinstated. At street prices of 50 CZK per half-litre of Pilsner, it was thus a punch in the face when the bartender at the airport bar asked for 145 CZK for the carelessly tapped beer, offering crap music and uncomfortable seating for the price. I refused the beer, because I prefered to keep only the good memories of the brew. NP: OSI: Blood

Yesterday s RAM mystery was solved after dozens of people have written in with helpful comments. Thank you all. I need to find time to create a proper, commentable blog soon so that you all can get the credit you deserve! I now have the full 4GB of RAM available:

$ free   head -2
            total       used       free     shared    buffers     cached
Mem:       4063324    1657440    2405884          0     149080     643732

Yay! The suggestions ranged from timing issues to defective modules, from 64bit OS via south/north bridge limitations to sacrificing chickens to god spongebobs. In the end, several people put their finger on a concept called memory hole remapping , and that turned out to be the ticket. Here is what I learnt: This motherboard and the CPU memory controller supports (which means can address ) 4096MB of RAM. Since extensions cards, such as the graphics card and the network controller, also need addressable slabs of memory for their RAM or registers, ranges in the upper memory regions are mapped to the RAM of those cards. The file /proc/mtrr gives a clue over what s happening:

reg00: base=0x000000000 (    0MB), size= 2048MB, count=1: write-back
reg01: base=0x080000000 ( 2048MB), size= 1024MB, count=1: write-back
reg02: base=0x0c0000000 ( 3072MB), size=  256MB, count=2: write-combining
reg03: base=0x0d8000000 ( 3456MB), size=  128MB, count=1: write-combining
reg04: base=0x0e8000000 ( 3712MB), size=   64MB, count=1: write-combining
reg05: base=0x0ec000000 ( 3776MB), size=   64MB, count=1: write-combining

Registers 0 and 1 represent the addressable RAM available to the operating system. At position 3072, we find two slabs of 256MB each, most likely corresponding to the two graphics cards in the system. I think that the 128MB in register 3 is the AGP aperture. Registers 4 and 5 correspond to 64MB slabs each, most likely relating to the two network cards I have installed. The result is that addresses in memory beyond 3072MB are routed to the extension cards, which overshadow the original system RAM and leave it unusable. When I turned on the memory hole remapping feature in the BIOS, I empowered the BIOS to remap the memory space. The way it does this is BIOS-dependent, but most likely, it will remap the overshadowed memory region to the end of the total memory space. This is a software-only remapping, but the effect is that the system appears to have more memory. /proc/mtrr now reads:

reg00: base=0x000000000 (    0MB), size= 4096MB, count=1: write-back
reg01: base=0x100000000 ( 4096MB), size= 1024MB, count=1: write-back
reg02: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachable
reg03: base=0x0c0000000 ( 3072MB), size=  256MB, count=1: write-combining

I am not entirely sure how to interpret this yet, but you can see that there are now an additional 1024MB at position 4096 (register 1), so the system seems to have 5GB of RAM available. Register 2 describes memory between 3072 and 4096 as not cacheable, because it maps to extension cards. I do not know how register 3 fits into the picture. By remapping, we cross the 4096MB mark. To make use of the additional memory, we will need PAE enabled to give you 36 bits, or a 64bit operating system: 2^32 is 4096M, so with only 32 bits available, 4GB is all the memory that can ever be addressed (dirty hacks excluded). Fortunately, this system has been running the Debian amd64 port even before it was officially released, and thus, once the BIOS remapped the memory, I could fully make use of it. Mystery solved, some knowledge acquired, many of you left to thank!

Update: The mystery has been solved. Thanks to everyone who wrote in. Despite the 4 1Gb RAM modules I added to the Asus A8V Deluxe mainboard of one of my machines, the BIOS (and Linux) only sees 3072 Mb. The BIOS is up-to-date (version 1018.002). The specs say that memory is

4 184-pin DIMM, Max. 4 GB, DDR 400/333/266 SDRAM, ECC, Non-ECC, Un-buffered
Dual Channel memory architecture

The four slots are labelled A1, A2, B1, and B2. A1 and B1 are blue, the other two are black. All four chips are DDR400 CL3, even though they are by different manufacturers. With all four slots filled, I get 3072 Mb of RAM. If I remove either of a black module, the RAM falls to 1024 Mb. If I remove either of a blue module, the RAM falls to 2048 Mb. Any idea what is going on? How can I get the full 4Gb I want? Please write in if you know.

An older machine of mine previously ran off 7 harddisks, which were added over the years. Initially, there were two 80Gb drives, then one failed and I replaced it with a 160Gb drive, leaving 80Gb of unused space. At some point I had two 120Gb drives, which I also added, and then another 160Gb and a 250Gb drive, and finally I replaced the first 160Gb with a 320Gb one. Between the different RAID1 and RAID5 arrays across these disks, as well as the LVM configuration atop, I had completely lost overview of the assembly. It didn t help that the drives were all IDE (PATA), so you can imagine the cable mess! When two drives started reporting S.M.A.R.T. errors, I took the opportunity to replace the mess with the four 250Gb SATA drives I had lying around at home. The problem was that two of the four SATA ports apparently didn t work. Hence I went to try a BIOS upgrade for the plain Asus K8V motherboard (Via VT8237 chipset). I don t remember what floppies look like, and I hate all these DOS-based flasher programs anyway, so this time around I gave flashrom a try (using the outdated version in Debian lenny). I managed to read out the existing ROM image, but even after verifying that it looked very similar to the new one, I was still scared of turning my computer into a brick. While waiting for people on the #flashrom/irc.freenode.org channel to reassure me, I managed to wipe the saved image and essentially had no other way but to move forward. I used flashrom to verify that the image was written and rebooted and the system came back up, this time with support for all four SATA drives. Thank you, flashrom people!

Your favourite pretend-do-no-evil information monopolist appears to have discovered wit: This is part of ongoing design experiments. What should happen is that the message fades out and gives way to the content that used to be there. Some of us continue to oppose JavaScript bling, or use text-mode browsers. In those, it just looks like Google inserted space between the search box and the money-making/legalese stuff at the bottom to be able to declare it intentionally blank. How witty! Apart from that, you cannot fade something else into the space you previously declared intentionally blank . Go Google. Be good!

Before I forget, or drown in a puddle of Guinness: I successfully defended my Ph.D. research and passed with a few recommendations for minor changes. I expect to have the thesis and data public around the end of the calendar year. Thanks to everyone for putting up with me, and for your support.

I am sitting in Lero, waiting for the next 30 minutes to pass before I shall defend my Ph.D. thesis in front of a board of three examiners. It will start at 9:30 GMT. I cannot decide whether I am nervous, or whether I had too much coffee. Someone just asked me if I am well-prepared Yes, I worked four years for this. Excited and prepared. If you want to tune in, my initial, public presentation will be webcast (using proprietary media, unfortunately (Flash required)). I hope to have a recording available later. The following interrogation by the examiners is not public (even though I wanted it to be). I forgot my Lederhosen at home, or I would have worn them today.

Today I branched the (LaTeX) source of my Debian book in preparation of the second edition, targetted for the forthcoming Debian 6.0 squeeze . I am very excited to come back to this project after deprioritising it in favour of my research for too long. Thanks to my publishers Open Source Press and No Starch Press for their patience. If you want to be the first to receive announcements about the book, such as release dates, previews, or other information, please sign up to the announcement mailing list. The second edition has two goals:

• Bring all existing content up-to-date with what has changed in Debian in the last four years.
• Add additional material, such as (but not limited to): SecureApt, backporting, using version control for packaging, IPv6, dm-crypt, Unicode, and virtualisation. If you have any suggestions or wishes for topics I should write about, let me know at feedback t debiansystem d t info.

I want to re-establish the book as the standard Debian reference, but also to give my existing readers a good reason to buy the book again. While we re on the topic of buying: yes, the book is pay-ware, and yes, in some ways that goes against the spirit of open-source, and Debian in particular. However, this is a choice I made. If this book didn t have a commercial basis, then I could not put as much time into it, and I would not be able to profit from the outstanding help from the folks at Open Source Press and No Starch Press it is thanks to their advice and experience that the book turned out to be as it is. The book will be available as PDF from the start, so if you care as much about the environment as I do, you ll prefer to get that over the Dead Tree Version. And if you re as haptic as I am, then you still have the option to buy the book so you can hold something real in your hands. I am looking forward to working on this project, to working intensely with the Debian community throughout, and to your input.

Today, Fabio and I extended the deadline of the call for papers of the upcoming Distro Summit until 18 October 2009. We have already received a number of great proposals, but not enough to complete the schedule. So if you have anything to say on the topic of distributions (such as, but not limited to the various Linux distributions), and especially on cross-distro collaboration, please launch your mailer and compose a message as detailed in the instructions. Distro Summit will take place as part of LCA 2010 in Wellington, New Zealand. Yes, that is on the Southern Hemisphere, and yes, that is far away from where many of you live, but it s well worth a visit, if you ll take my word for it. Unfortunately, we are unable to provide sponsorship for speakers, so talk to your employer or your distro now and get the ball rolling. You have a little more than a week to get your bids in! What are you waiting for? This is a once-in-a-lifetime chance: Distro Summit, LCA, and Wellington, all-in-one. It doesn t get much better than that. Let s hear your proposal! PS: I loved Wellington (and its people) so much that I picked up my very own pet Wellingtonian!

If you have recently tried to subscribe to one of my mailing lists but could not confirm the subscription, please try again. I just found and fixed a silly bug in Mailman. The bug might have prevented people from subscribing to any of my lists, because the e-mail confirmation handling was broken. I am kind of surprised that this has not been spotted earlier, especially because it s such a simple and obvious fix. I suppose I must be among the only people using postfix-to-mailman.py as a gateway between postfix and Mailman. NP: Led Zeppelin: Physical Graffiti

At 14:45 CEST, today, 1 October 2009, I submitted my thesis to the print shop. Sarah has kindly agreed to pick it up tomorrow and drop it off at Brian s office, from where it will go to the three examiners. On 30 October at 9:30 IST, I will stand to publicly defend my work at the University of Limerick. I expect to receive a list of changes to be made from the examiners, hopefully minor, such that there won t be much in the way before I submit the final version to the University Board by 3 December. Once they give their green light, I will make it available on my research website, along with the data. I expect this to happen in January 2010. I have many many people to thank and will do so in due time. Right now I just want to thank Brian Fitzgerald and Kieran Conboy for their flexibility. I was supposed to have the thesis done by the end of August to given them a month to read it over, but by then, I only had the first six (of eight) chapters done. We hence spent all of September sending back and forth PDFs and paper comments, working iteratively so that today, I could put the finishing touch on the work and keep the schedule. The next thing on my agenda is the second edition of my book, which we are targetting for Debian squeeze . If you would like to be the first to receive more details, sign up to the announcement mailing list.

This coming Sunday, Germany is voting its parliament for the next four years. I wanted to participate remotely, but they didn t let me. The letter, inviting me to order the voting forms, was sent too late. Instead of ordering and waiting for the voting forms, hoping for them to arrive back in Germany before the municipal agency stops all their work at 16:59 on Friday afternoon, I filled in the fields in a way to give my mother full authority to submit my vote. On the document, it only said that she would need a notary procuration, which she has. When my mother tried to vote in my name, they refused her: the vote has to be confidential, and proxies may only vote for disabled people, or those who cannot read. I can understand half of the motivation behind this it guards against abuse but the bottom line is that I am now unable to cast a vote. Democracy stabbed by bureaucracy. Well, democracy to the limited degree that we have it in Germany. I do not mean to criticise at this time that voting in Germany means to place trust into people you d rather not trust, but once they get majority, they can do whatever they want for four years. The specific shortcoming in our democracy on which I would like to focus is the five percent clause , which states that a party needs to reach 5% of votes before they can enter the parliament; if those 5% are not reached, all votes for the party are basically lost. I suspect that this clause is what s keeping back a lot of people from electing Germany s pirate party, which focuses on the importance of freedom in today s media-centric and interconnected world. It seems doubtful that the Piratenpartei will reach the limit this round, but given all the attention they recently received, as well as their plain cause, I would not be surprised if they did. I think it is important for people to vote the party they want, without letting the 5% limit discourage them. Anything else would be undemocratic. Let s return to the pirates: it s paramount for the government to consider freedom and related issues, and to put clueful people on the job. The Piratenpartei is the only party that approaches the issue properly, all the others are just too happy to succomb to castration of personal freedom for such lofty goals as anti-terrorism, or the containment of child pornography. I don t like terrorists, and I think child pornography is among the most horrific ways to abuse defenceless humans in irrevocable ways that I can imagine. But I also don t like losing my deodorant to airport security, cameras tracking my every move, (diletantic) censorship on the Internet, and lack of control over my own data. Time to put an end to that, and to solve problems without castrating freedom. Benjamin Franklin is credited with saying: they that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. The Piratenpartei is probably not capable of leadership at this point, for they are young. But they can help the leading party maintain a bearing. That s what the Greens did, successfully albeit slowly, and I appreciate that the pirates are approaching the matter before it will be too late. The media have not missed a chance to try to stomp over them in recent months, but at the end of the day, after all the slander and some obvious mistakes on the side of the party, I still see a group of people prepared to fight for freedom with an open and honest approach. The run-ins with the media are hardly about content, it seems more that the Piratenpartei is simply unable to become part of the sensationalist mash which is our media-politician ensemble. And I think that s good. I appreciate a party communicating not via their PR office, but with new media, like blogs, or e-mail. I appreciate that their people maintain the personality of a human trying to do politics, not a politician trying to come across as human. If my municipality had let me vote, I would have selected the pirates. They may not make the 5% this year, but I would not let that be a reason against them. If one believes in democracy, then it s important to do it right.

Just thought I should let you all be part of this: I love my awesome girlfriend once for every byte ever pushed onto the Web two-point-oh. One could say that the complexity of this love is (n). NP: Porcupine Tree: The Incident

If you deal with SMTP servers, you probably know swaks, the Swiss Army Knife for SMTP . Great tool for anything related to sending mails. Today, I found its counterpart for receiving mails:

python -m smtpd -n -c DebuggingServer localhost:1025

that binds a no-frills SMTP server to the specified socket which does nothing but talk SMTP to connecting clients and print received messages to stdout, e.g.:

---------- MESSAGE FOLLOWS ----------
Date: Fri, 22 May 2009 21:21:40 +0200
To: madduck@madduck.net
From: madduck@lotus.madduck.net
Subject: test Fri, 22 May 2009 21:21:40 +0200
X-Mailer: swaks v20061116.0 jetmore.org/john/code/#swaks
X-Peer: 127.0.0.1
This is a test mailing
------------ END MESSAGE ------------

Sweet, very sweet! NP: Porcupine Tree: Signify

Take the following URL:

http://my.server.tld/wow/summary/2009/05/21/?style=boxed

and imagine the following Python function:

def summary(request, year, month, day):
    response = HttpResponse()
    // do something
    return response

If you see a relation between the two, then Django is for you. The missing part is a URL mapping, a list of regular expressions, which transforms a URL into a keyword list, and maps it to a function (a HTTP view, as it s called):

(r'^wow/summary/(?P<year>\d 4 )/(?P<month>\d 2 )/(?P<day>\d 2 )/$',
    mysite.wow.summary,  'additional':'information' )

You do not have to use keyword arguments, positional parameters will work as well, and you can do anything you like within the powers of regular expressions, but the result is always along the same lines: When someone visits the above URL, Django calls the above summary( ) function (defined in the modules mysite/wow.py) and passes an object encapsulating the request, along with the components from the URL as parameters:

summary(request, 2009, 05, 21,  'additional':'information' )

It then renders the return value. The function can also throw an exception if it need to communicate e.g. a 404 error. Very clean, and there are even generic views if your data doesn t need any massaging, of if you just want to do standard things like generate PDFs. This is the core idea of Django, which I ve gathered from one talk and four hours of invested time. It s a truly thin layer, and its elegance made me bouncey. The QUERY STRING of the request will be passed in the request object, along with all the other CGI variables you re familiar with. Django provides a large number of helpers and shortcuts to save you from having to do the ugly work, including topics such as internationalisation, syndication, authentication, file uploads, and caching. In addition, Django gives you an object-relational_mapper to map the data into, and manage the data definition within your favourite RDBMS, a templating language designed to fit in with the whole philosophy, and widgets to work with forms. You do not have to use any of those. You are free to use any other Python module for the task, just as you can use all of the other features of and modules written for Python. This is particularly powerful in the context of the middleware layer. There s also a large number of pluggable applications and snippets for re-use. The only concern I have at this point is whether features like tagging are going to find their way into the framework, because tracking numerous external plugins for a site and keeping them working across versions reminds me of the nightmares I had with Zope and Plone (I stopped working with (and on) those before I had a chance to dive into version 3 of each). The fact that Django leaves data storage to other tools (in fact, it s entirely up to you) makes those nightmares seem further distant. All in all, I am excited to have had a chance to take a brief glance at the software, and I am looking forward to doing more with it. Unfortunately, it looks like it will take quite some time to wade through existing plugins and design the rest of an application that can finally replace the horrific dung-pile that is my homepage.

In Myanmar, Aung San Suu Kyi and 2 000 monks and activists are being detained, as the miltary junta in the country presses new, ridiculous charges against Suu Kyi a few days before the end of her house arrest. Suu Kyi, recipient of the Nobel Peace Prize and proponent of democracy, is leader of the opposition and the greatest threat to the junta s grasp on power. Given the inhumane conditions imposed upon the Burmese, and the ruthless slaughter of those who do not play by the military s rules, it is not hard to imagine that she would win any fair election, as soon as she would get a chance to run. However, if the junta are successful in pushing charges Suu Kyi is accused of breaching the house arrest after an American man sneaked into her house uninvitedly she could be locked up until after the elections in 2010, effectively disarming the threat she poses to the military regime. She is threatened to be kept in the notorious Insein Prison without medical care. As she is suffering from serious illness, this puts her life at risk. I do not usually campaign for political topics, but I ve been to Myanmar and this issue touches me deeply. The people in Myanmar need our support. Even though the 2007 Burmese anti-government protests seriously weakened the military and brought hope to the lives of millions that have been oppressed for years, the events quickly vanished from the media and allowed the junta to regain control. We must not let that happen again. It is of utmost importance to show our support and not look away. One way of doing so is by increasing visibility. If you care, please write to your newspapers, publish on your blog, and do everything else you can to raise awareness. Another way to help the Burmese people is through signing the avaaz.org petition to UN Secretary General Ban Ki-moon. Best to do all of the above! Unfortunately, avaaz.org force-subscribes everyone who signs a petition to their newsletter, which may well not be what you want. However, please do not let it stop you from signing the petition. You can later unsubscribe from the (infrequent) newsletter. I have informed the staff members of this concern. If you would like to voice your opinion too, write to info t avaaz dot org and encourage them to make the newsletter subscription optional. But make sure to also sign the petition, please.

Some might dread the feature of read notifications supported by certain MUAs; some call it an invasion of privacy ; and yet, it can also be useful in certain situations: When a message is read or seen in a MUA supporting this extension, the programme emits a notification back to the sender saying something along the lines of your message was read on . This is good to know, especially in times when you cannot wait for the failed-delivery-notification that follows four to five days of unsuccessful (but furious) attempts of some delivery agent, assuming it doesn t get trashed as spam. Such a read notification is logically a reply to the original message, isn t it? The RFC 680 proposed in April 1975 defines the header References as a way to point to other correspondence which this message references . This header, along with In-Reply-To (defined in the same RFC), is commonly used in every-day mail traffic to refer to previously exchanged messages, and enables mail readers to thread separate messages together into coherent conversations (it takes a human to remove the coherence, the technical aspect is infallible). Cut. Microsoft was also founded in April 1975, and it took them 20 years to barely manage to squeeze through the Internet door without the proverbial foot in it. They published a browser and several e-mail programmes, and it always appeared as if they tiredlessly tried to be different from the rest, attempting to form a clique of users, a Microsoft league in which to increase their revenue through network effects. Sounds bad, is bad, but yet again, they managed, through unimaginable feats of entrepreneurial genius and ruthless behaviour. Cut. In 1982, STD11 declared the aforementioned In-Reply-To and References headers as standards. At that time, Microsoft software didn t even know what a computer network was. Cut. Does it come as a surprise that read notifications sent by Microsoft e-mail programmes, such as Microsoft Outlook do not make use of either of these standard headers to tag read notifications they send? Instead, Microsoft pushes Thread-Topic and Thread-Index, which are undocumented and thus probably only work in a Microsoft-only context. How am I supposed to assume anything else than Microsoft actively trying to oppose standards. Anyone who boycotts standards is hindering progress and should be left behind. It s good to see that the Internet society seems to follow that trend more and more. Update: I found a way to extract the data to recreate the In-Reply-To with procmail. I don t see a way to do the same for the References header. Also, I ve only verified that this works for message disposition notifications from Outlook 2003, although I expect it to work for other, similarly crippled MUAs too.